The invention relates generally to systems and methods that pertain to interactions between a vehicle and an occupant within the vehicle. More specifically, the invention is a system or method for enhancing the decisions (collectively “decision enhancement system”) made by automated vehicle applications, such as safety restraint applications.
Automobiles and other vehicles are increasingly utilizing a variety of automated technologies that involve a wide variety of different vehicle functions and provide vehicle occupants with a diverse range of benefits. Some of those functions are more central to the function of the vehicle, as a vehicle, than other more ancillary functions. For example, certain applications may assist vehicle drivers to “parallel-park” the vehicle. Other automated applications focus on occupant safety. Safety restraint applications are one category of occupant safety applications. Airbag deployment mechanisms are a common example of a safety restraint application in a vehicle. Automated vehicle applications can also include more discretionary functions such as navigation assistance, and environmental controls, and even purely recreational options such as DVD players, Internet access, and satellite radio. Automated devices are an integral and useful part of modern vehicles. However, the automated devices embedded into vehicles need to do a better job of taking into account the context of the particular vehicle, and the person(s) or occupant(s) involved in using the particular vehicular user. In particular, such devices typically fail to fully address the interactions between the occupants within the vehicle and the internal environment of the vehicle. It would be desirable for automated applications within vehicles to apply more occupant-centric and context-based “intelligence” to enhance the functionality of automated applications within the vehicle.
One example of such omissions is in the field of safety restraint applications, such as airbag deployment mechanisms. Airbags provide a significant safety benefit for vehicle occupants in many different contexts. However, the deployment decisions made by such airbag deployment mechanisms could be enhanced if additional “intelligence” were applied to the process. For example, in several contexts, the deployment of the airbag is not desirable. The seat corresponding to the deploying airbag might be empty, rendering the deployment of the airbag an unnecessary hassle and expense. With respect to certain types of occupants, such as small children or infants, deployment of the airbag may be undesirable in most circumstances. Deployment of the airbag can also be undesirable if the occupant is too close to the deploying airbag, e.g. within an at-risk-zone. Thus, even with the context of a particular occupant, deployment of the airbag is desirable in some contexts (e.g. when the occupant is not within the at-risk-zone) while not desirable in other contexts (e.g. when the occupant is within the at-risk-zone). Automated vehicle applications such as safety restraint applications can benefit from “enhanced” decision-making that applies various forms of “intelligence.”
With respect to safety restraint applications, such as airbag deployment mechanisms, it is useful for automated applications to obtain information about vehicle occupants. With respect to airbag deployment mechanisms, the existing art typically relies on “weight-based” approaches that utilize devices such as accelerometers which can often be fooled by sudden movements by the occupant. Vehicle crashes and other traumatic events, the type of events for which safety applications are most needed, are precisely the type of context most likely to result in inaccurate conclusions by the automated system. Other existing deployment mechanisms rely on various “beam-based” approaches to identify the location of an occupant. While “beam-based” approaches do not suffer from all of the weaknesses of “weight-based” approaches, “beam-based” approaches fail to distinguish between the outer extremities of the occupant, such as a flailing hand or stretched out leg, and the upper torso of the occupant. Moreover, “beam-based” approaches are not able to distinguish between or categorize different types of occupants, such as adults versus infants in baby chairs versus empty seats, etc. It may be desirable for vehicle safety applications (and other applications that would benefit from obtaining occupant information) to obtain both location information (including by derivation, velocity and acceleration) about the occupant as well as information relating to the characteristics of the occupant that are independent of location and motion, such as the “type” of occupant, the estimated mass of the occupant, etc. It may be desirable for decision enhancement systems in vehicles to utilize an image of the occupant in obtaining contextual information about the occupant and the environment surrounding the occupant. Although the existing art does not teach or even suggest an “image-based” approach to safety restraint applications, an “image-based” approach can provide both location information as well as occupant categorization information.
Image processing can provide increasingly useful possibilities for enhancing the decision-making functionality or “intelligence” of automated applications in vehicles and other applications. The cost of image-based sensors including digitally based image-based sensors continues to drop. At the same time, their capabilities continue to increase. Unfortunately, the process of automatically interpreting images and otherwise harvesting images for information has not kept pace with developments in the sensor technology. Unlike the human mind, which is particularly adept at making accurate conclusions about a particular image, automated applications typically have a much harder time to correctly utilize the context of an image in accurately interpreting the characteristics of the image. For example, even a small child will understand that person pulling a sweater over their head is still a person. The fact that a face and head are temporarily not visible will not cause a human being to misinterpret the image. In contrast, an automated device looking for a face or head will likely conclude in that same context that no person is present. It would be desirable for decision enhancement systems to apply meaningful contextual information to the interpretation of images and other forms of sensor readings. One way to apply a meaningful context for image processing is to integrate past images and potentially other sensor readings into the process that evaluates the current or most recent sensor readings. Past determinations, including past determinations associated with probability values or some other form of confidence values can also be integrated into the decision making process. The use of Kalman filters can provide one potential means by which the past can be utilized to evaluate the present.
Another obstacle to effective information gathering from images and other forms of sensor readings is the challenge of segmenting the focus of the inquiry (e.g. the “segmented image” of the occupant) from the area in the image that surrounds the occupant (e.g. the “ambient image”). Automated applications are not particularly effective at determining whether a particular pixel in an image is that of the occupant, the vehicle interior, or representative of something outside the vehicle that is visible through a window in the vehicle. It can be desirable for a decision enhancement system to apply different segmentation heuristics depending on different lighting conditions and other environmental and contextual attributes. It may also be desirable for a decision enhancement to utilize template or reference images of the vehicle without the occupant so that the system can compare ambient images that include the occupant with ambient images that do not include the occupant.
The varieties of occupant behavior and vehicle conditions can be voluminous, and each situation is in many respects unique. Such a divergent universe of situations and contexts can overwhelm vehicle applications and other automated devices. It would be a desirable approach to define various conditions, modes, or states that relate to information that is relevant to the particular context. For example, with respect to safety restraint applications, the occupant within the vehicle can be considered to be in a state of being at rest, in a state of normal human movement, or in a state of experiencing pre-crash breaking. It can be desirable for the decision enhancement system to associate a probability with each of the predefined conditions in making decisions or applying intelligence to a particular situation. For example, in the context of a deploying airbag, it can be desirable for the decision enhancement system to calculate probabilities that the occupant is in a state of pre-crash breaking, is asleep, or is riding normally in the vehicle.
Various cost-benefit tradeoffs preclude effective decision enhancement systems in vehicles. For example, standard video cameras do not typically capture images quickly enough for existing safety restraint applications to make timely deployment decisions. Conversely, specialized digital cameras can be too expensive to be implemented in various vehicles for such limited purposes. It would be desirable for the heuristics and other processing applied by the decision enhancement system to generate timely “intelligence” from images captured from standard video cameras. This can be accomplished by focusing on certain aspects of the image, as well as by generating future predictions based on past and present data. An approach that attempts to integrate general image processing techniques with context specific vehicle information can succeed where general image processing techniques would otherwise fail.
The solutions to the limitations discussed above and other limitations relating to automated vehicle applications are not adequately addressed in the existing art. Moreover, the existing art does not suggest solutions to the above referenced obstacles to decision enhancements. The “general purpose” nature of image processing tools and the “general purpose” goals of the persons developing those tools affirmatively teach away from the highly context-specific processing needed to effectively enhance the decision-making of automated vehicle safety restraint applications.